Evaluation Methodology for a Telephone-Based Conversational System

This paper describes a suite of metrics that we use to evaluate our JUPITER system which provides worldwide weather information over the telephone. Since May, 1997, we have made the system available to the general public via a toll-free number and have collected approximately 35,000 utterances to date. These data have proven invaluable for system development and evaluation. Because JUPITER makes use of many component spoken language technologies, we evaluate each as a stand-alone system. However, JUPITER must also be evaluated as to how it accomplishes its main goal, that of understanding user queries and providing an informative response. This requires an evaluation that takes into account how these various technologies t together in an end-to-end fashion. We will describe an automated end-to-end evaluation system we have implemented and compare it to a subjective evaluation conducted by hand.